{
  "cells": [
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/pinecone-io/examples/blob/master/learn/generation/llm-field-guide/falcon/falcon-40b-chatbot.ipynb) [![Open nbviewer](https://raw.githubusercontent.com/pinecone-io/examples/master/assets/nbviewer-shield.svg)](https://nbviewer.org/github/pinecone-io/examples/blob/master/learn/generation/llm-field-guide/falcon/falcon-40b-chatbot.ipynb)"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "JPdQvYmlWmNc"
      },
      "source": [
        "# Falcon 40B Chatbot in Hugging Face and LangChain\n",
        "\n",
        "In this notebook we'll explore how we can use the open source **Falcon-40B-Instruct** model in both Hugging Face transformers and LangChain.\n",
        "At the time of writing, this was the top ranked open source LLM according to the [Hugging Face Open LLM Leaderboard](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard).\n",
        "\n",
        "---\n",
        "\n",
        "\ud83d\udea8 _Note that running this on CPU is practically impossible. It will take a very long time. If running on Google Colab you go to **Runtime > Change runtime type > Hardware accelerator > GPU > GPU type > A100**. Using this notebook requires ~28GB of GPU RAM._\n",
        "\n",
        "---\n",
        "\n",
        "We start by doing a `pip install` of all required libraries."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "K_fRq0BSGMBk",
        "outputId": "7e62e4c5-a698-4343-d6af-fb6399ef8156"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\u001b[?25l     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m0.0/7.2 MB\u001b[0m \u001b[31m?\u001b[0m eta \u001b[36m-:--:--\u001b[0m\r\u001b[2K     \u001b[91m\u2501\u2501\u001b[0m\u001b[91m\u2578\u001b[0m\u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m0.5/7.2 MB\u001b[0m \u001b[31m55.6 MB/s\u001b[0m eta \u001b[36m0:00:01\u001b[0m\r\u001b[2K     \u001b[91m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m\u001b[90m\u257a\u001b[0m\u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m5.2/7.2 MB\u001b[0m \u001b[31m80.7 MB/s\u001b[0m eta \u001b[36m0:00:01\u001b[0m\r\u001b[2K     \u001b[91m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m\u001b[91m\u2578\u001b[0m \u001b[32m7.2/7.2 MB\u001b[0m \u001b[31m86.2 MB/s\u001b[0m eta \u001b[36m0:00:01\u001b[0m\r\u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m7.2/7.2 MB\u001b[0m \u001b[31m63.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m227.6/227.6 kB\u001b[0m \u001b[31m33.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m42.2/42.2 kB\u001b[0m \u001b[31m6.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m1.2/1.2 MB\u001b[0m \u001b[31m75.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m109.1/109.1 MB\u001b[0m \u001b[31m14.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m97.1/97.1 MB\u001b[0m \u001b[31m17.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m268.5/268.5 kB\u001b[0m \u001b[31m37.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m7.8/7.8 MB\u001b[0m \u001b[31m81.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m1.3/1.3 MB\u001b[0m \u001b[31m75.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m90.0/90.0 kB\u001b[0m \u001b[31m15.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m49.1/49.1 kB\u001b[0m \u001b[31m7.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h"
          ]
        }
      ],
      "source": [
        "!pip install -qU transformers accelerate einops langchain xformers bitsandbytes"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "VHQwEeW9Zps2"
      },
      "source": [
        "## Initializing the Hugging Face Pipeline"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "mElf068NXout"
      },
      "source": [
        "The first thing we need to do is initialize a `text-generation` pipeline with Hugging Face transformers. The Pipeline requires three things that we must initialize first, those are:\n",
        "\n",
        "* A LLM, in this case it will be `tiiuae/falcon-40b-instruct`.\n",
        "\n",
        "* The respective tokenizer for the model.\n",
        "\n",
        "* A stopping criteria object.\n",
        "\n",
        "We'll explain these as we get to them, let's begin with our model.\n",
        "\n",
        "We initialize the model and move it to our CUDA-enabled GPU. Using Colab this can take 5-10 minutes to download and initialize the model."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000,
          "referenced_widgets": [
            "dcd114a0c813495ba1b4fff083822911",
            "bb29ff54a20f441088d232604e488a15",
            "4014f7dd2c1b4f04a5f46812529b4199",
            "0f6deac556584e7586d3a6dd570ac99c",
            "f084f4404b0046c4aa996cd5d0b28b4f",
            "15fe151e2e114afa84e1c05726dd83aa",
            "460f2523c933410d9b7772b7fdc68928",
            "9111a0a9605c43b593078bbfc3df8118",
            "1b8a8143f4fa403e8d6fc290a3750d88",
            "810f0a5a0f264c6eb4ed10d651022202",
            "311f8b36cc2244fdb7abbf2b850aed6c",
            "a42f0b74e8714faeaa2cb4ebd8a0b292",
            "ca8901174bd348428ffb2b4b3dd79939",
            "2a6923dc535b4c8d869326811c1b33e4",
            "f119ba10b71f495dbb7da4f28888e850",
            "5178cf705e6647229c3466bd86c13148",
            "da35e82b6d784628bb5e4505c24ff9e3",
            "71e1beb8a02b449896e4bce2c3c92aad",
            "d3943ef69ccf4d6f834073a37c45efba",
            "799e1460f1714e25af77eda809cc6700",
            "51e451d2242b4d9b90939b5683f456ea",
            "195301bde7e24c54a951a045421c9329",
            "03e9aeb80e12424db0e845a92b202257",
            "d2dd2d3cb933425482c89acd7f4916ad",
            "c62862eac048444e8e41c9e15eb622c2",
            "7b8d56b74a9d42beb3fd66030855b48b",
            "dd23f07c6393428686ed62eb0cb34a32",
            "5a185d9e39d44f22a8b7059ec688512f",
            "b0eeb4d305644f3ea99ce6f92095a044",
            "cf9e98de503f4c59875f48980e29486b",
            "c5925d90f45d4d578d8d9ba0b20e80bc",
            "1b717742f34e442083f9445f931740f5",
            "006b9c26cd3b45d59cd41ea11606d30f",
            "5baa0f9ed9ea40b4b0b3e2a87e085990",
            "c9194bb7fc764c45b5bd40880a086392",
            "7ab6144df31e4048a95aa07b66d21412",
            "ed6c7d35eb594018830241fbc13bb36e",
            "c9fd93cc441b43f2b3ea5ca6f03104e7",
            "12754ad460e14885bab25f3d04e598d4",
            "8485a5e2c5ce4b8ab3e27a7403f575ce",
            "ee2533bd1f4e4e02b65c466d355702c6",
            "a748540486af4718ba5b39d448c38c03",
            "f91f4b28c7434eaaba18260c73782d58",
            "7473add371094e6d9ae60ce40490ff80",
            "3576555eb6734dcb9ef9ec2703e7c147",
            "753f033c7c224f0a918d90ae125f467f",
            "cf53401f3c334ff0ad13198e949a337b",
            "7d837344814d418c89b08a5665c36937",
            "bdcc37e3b4e6421c86d0272034ee302d",
            "11c36b2d232149e5af74d6349642c9e1",
            "ff565eb2e7fc4ed1ba941e72c939cff3",
            "6cb518544ed74e18bfd95410d7a0729e",
            "e787246394374d84add20fde614ba459",
            "7ad3532a745b4914b6d5c87710bf9eee",
            "1c095d2ce29343c3a699adff4f0aad9e",
            "942c979b1b57465fb57fe18fc3bc9cec",
            "82cb277dae474d979c190573aed72ab6",
            "9198945395ce4473ab6619d3b7c991b0",
            "6bf2ce863c9a42248225f013bc0ef472",
            "931d8a102cba4359bd505587fad3c91d",
            "d8b4927b493b416f8aa15f6fee6845f4",
            "6d67927b5fe544b59ea49ee5fcd33c9a",
            "008b946b12e948a48fba589ef3983aa3",
            "4f173fee30164214ae8d04c18046bf15",
            "608b23b4013d453faa0a87a527a98fd6",
            "22a89e9ecdc140e39759b35e842f9728",
            "ba6101b5589c4a86a57f338cacbc9cec",
            "a2ccb3c4cdb840fd8ddd75c92e8ede8a",
            "b13fca47f48f49118cf3cda52d1a9fb9",
            "edbe12452c6d4ab7bb091aa6a5ad4c98",
            "5441021ce6e440cca82f846afd85f751",
            "4148e657eca64e87b14648368e33384e",
            "21fb1c8cfe3542d591e7ec564f2688b6",
            "effe9518cd3947ceaacfcc0108b58839",
            "b230a229680d4d078b31b93a330afca2",
            "be48fcc997f2469e978ae39324e3df12",
            "f3ed7c3124234a42800cc2a07467d15c",
            "7388b76763c24860bf69981d883285dc",
            "f3c3ef087e6b4a19b4ca9a6848abdae4",
            "a9213d1b4a7b46a19a822ace027176f2",
            "6769cb16d3bf486aa198debaa4cdd31a",
            "8452692026fc486f9f265e5d39d42c25",
            "8ab2f6c030f74f1686e2a5c8b048c3c2",
            "e3e045a17e884af5a47ca03bf10bafea",
            "48a6e76c66c74689b8095d0ef17f8a2e",
            "43fc75c7f08f4670af028b7da3e0857b",
            "c31b8ee33d094d9ab1a355209aba4260",
            "8a31f51cf0084be7869f59ad4c9b6b45",
            "be04aed8052745f1be93f37574a8d998",
            "0078ed5424194ae180df34d3a8d596ee",
            "abb2fd2d1f7b488dbbfa18828d2c229a",
            "484c1dea56de454b88ecd350c1915315",
            "7996b492973849dcb253e8bae3e4f409",
            "da8cd815eb2f4db685da1a3eb82c3a15",
            "cdaec79c140c430cb2f4e0725a96eb7a",
            "6eb5d6f0cc6e4875a04f342e9ec8247c",
            "0aa8c47f7851471caf0fe69e54b53e30",
            "e7f89fd1b6484378ac60c484e0b49a5b",
            "94dd1b82d05f4a529a3917f623923fdc",
            "67ba6271f2d841a989ec53ceabc53401",
            "8ab9021255294d65a9026e71b7de68c9",
            "1c7a3f5216e44f0d930bb01672ac74e4",
            "1bc5b6188e8448fdaef802ef227584f1",
            "49557ed7c0bb481690f4c98ae68c547f",
            "3d76dc6cc94f4b98a29dc43015bde3b9",
            "9309831492f74b06b95266c0fbe7eb22",
            "ae97ab542321401f8a77ac23745bd83f",
            "050184a69c014c2aac5832ea7265a947",
            "993a7dab73914041a9f99ed6e841187f",
            "6dcad84bf429425195fd5de0dd6affb3",
            "441a0b81e07d4db5b71b2c4cb01704b8",
            "da107a144e234a0f841c0a991c3e2745",
            "50bc986853c149f4a665ddf769f79b5d",
            "60592100afc9454981c608dfe27ff212",
            "a0a36f182a844ad5a1dc95ff8174f547",
            "d9d17e62ec0a4fa0b907469d95231070",
            "5ac8db36ae8e4720b7f09d3fde3eb3f7",
            "906852fb8da84f21b59288ab04eea088",
            "fb7d390fb3eb43cd8ca2833359896f09",
            "1941428d9c344658ab6f6f5fcf6c0164",
            "3d4ee83cca8a41389f3730f0403334ba",
            "b46e2cd523de482ca1749d92f2370270",
            "39fc62d60a924bb49c08c77707c178d6",
            "638dd4e46df641c3a6bf2835a224aa45",
            "966f2ecb28a24272ba113b22848c021e",
            "4c2ed8a71cdb4d5a8fc8c4f8e14063e7",
            "bd2778262c0543f1ad9c25fee5153cfb",
            "dbb147aac4864ed9873a6bb01147c6df",
            "bbe227c5ffbe47e293b20269c6bf1962",
            "49de4428292d480caeabfb7c46cdb558",
            "53877755b7b548b3b281fc63a7ce80c6",
            "67a80db57098433aa471ce000c7fd121",
            "fb642fae86be40708b21552c8cf059d3",
            "31f5a16a9b2b47f5a805a674e23fd777",
            "dac797bad0874fe999a3b6d78588ec04",
            "e66e8d3e9bca419c89e6d5575e0bad4f",
            "879b82f7e4cc4212865394f1a0a48898",
            "cb5875cc14c94b6fa156eb1eff501990",
            "9f85a572c1c94474ad2872e763a2c8d1",
            "7a1a124af4fe4dbc945ed30f0ecebfe2",
            "36a83a7a250c463ba88f756bfed7c7bb",
            "0c19f788c0aa411b8a96a5f83d712b3b",
            "f7556aa731284183b1991e3522288c15",
            "b13699516caa45858d35baf312ccd80b",
            "a6820db2be134aa2ac77124bac47c78d",
            "83cc427331194e4d8ff56abede64a46f",
            "36532b22b5a945e1a95e1edd81dfbd7b",
            "c3c4fa7af5844d409af6457dce730e05",
            "e3a45aaf756b4852827b3be88e6929aa",
            "29b808f166f84493823355a9fdc833b7",
            "916c430ee06741789f4ed6b305f8e8cb",
            "39278f6f656a4d4d9b962b0178b51018",
            "663087fbb5a7470f874193230e813a77",
            "689f72a9baaf4f50b894fd46bceb1455",
            "f9c8ec99ed3e4e75a56a3a092a6db1a7",
            "40d956ad33094f6d9587a0e0a8144e08",
            "5075ee94c7fe456a9a82e4756abe3733",
            "6749a9428f3e44d2b35ae88ef061989a",
            "5f4a59fa92f442a8b48cf73de4dcdd49",
            "47a306768f1844ad92332a44a316620c",
            "61c0e0db34db458dbdd484d71d5f1524",
            "96a78e56e8a448348da11bea4f8ac3fd",
            "eb3edfbe4fe84496b7561b478321fa36",
            "07d18ac0693449e58db841ff94cdfc79",
            "cbb582a75e1e4a63be06e6c9ff6eb2da",
            "31a13483277042979c721a9a1581cfd9",
            "62897025239a4823904a7ad039115f98",
            "d243140f68fb4e1ab0a220b971c3bf12",
            "41a7c43846984d47b08a7844861d9606",
            "811b623d97f24e57b013f4ec9a487774",
            "d30d50293579484b8f96c3cd47500dc0",
            "fe3f3f4d09b04ee6b6a67e4570d07813",
            "32ca738255464e7d81f6ccf485b1effd",
            "dea33dc8c1d04310b9125cde24626513",
            "c761e3001627428a924d6d5a5f618767",
            "42c17130d7fe40aa994066c74a4f78c6"
          ]
        },
        "id": "ikzdi_uMI7B-",
        "outputId": "d890e809-4cbe-4f86-b37a-fa1ffe11061b"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "dcd114a0c813495ba1b4fff083822911",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading (\u2026)lve/main/config.json:   0%|          | 0.00/658 [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "a42f0b74e8714faeaa2cb4ebd8a0b292",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading (\u2026)/configuration_RW.py:   0%|          | 0.00/2.51k [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "A new version of the following files was downloaded from https://huggingface.co/tiiuae/falcon-40b-instruct:\n",
            "- configuration_RW.py\n",
            ". Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision.\n"
          ]
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "03e9aeb80e12424db0e845a92b202257",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading (\u2026)main/modelling_RW.py:   0%|          | 0.00/47.1k [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "A new version of the following files was downloaded from https://huggingface.co/tiiuae/falcon-40b-instruct:\n",
            "- modelling_RW.py\n",
            ". Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision.\n"
          ]
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "5baa0f9ed9ea40b4b0b3e2a87e085990",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading (\u2026)model.bin.index.json:   0%|          | 0.00/39.3k [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "3576555eb6734dcb9ef9ec2703e7c147",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading shards:   0%|          | 0/9 [00:00<?, ?it/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "942c979b1b57465fb57fe18fc3bc9cec",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading (\u2026)l-00001-of-00009.bin:   0%|          | 0.00/9.50G [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "ba6101b5589c4a86a57f338cacbc9cec",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading (\u2026)l-00002-of-00009.bin:   0%|          | 0.00/9.51G [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "7388b76763c24860bf69981d883285dc",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading (\u2026)l-00003-of-00009.bin:   0%|          | 0.00/9.51G [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "be04aed8052745f1be93f37574a8d998",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading (\u2026)l-00004-of-00009.bin:   0%|          | 0.00/9.51G [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "67ba6271f2d841a989ec53ceabc53401",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading (\u2026)l-00005-of-00009.bin:   0%|          | 0.00/9.51G [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "441a0b81e07d4db5b71b2c4cb01704b8",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading (\u2026)l-00006-of-00009.bin:   0%|          | 0.00/9.51G [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "b46e2cd523de482ca1749d92f2370270",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading (\u2026)l-00007-of-00009.bin:   0%|          | 0.00/9.51G [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "fb642fae86be40708b21552c8cf059d3",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading (\u2026)l-00008-of-00009.bin:   0%|          | 0.00/9.51G [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "b13699516caa45858d35baf312ccd80b",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading (\u2026)l-00009-of-00009.bin:   0%|          | 0.00/7.58G [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "===================================BUG REPORT===================================\n",
            "Welcome to bitsandbytes. For bug reports, please run\n",
            "\n",
            "python -m bitsandbytes\n",
            "\n",
            " and submit this information together with your error trace to: https://github.com/TimDettmers/bitsandbytes/issues\n",
            "================================================================================\n",
            "bin /usr/local/lib/python3.10/dist-packages/bitsandbytes/libbitsandbytes_cuda118.so\n",
            "CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching in backup paths...\n",
            "CUDA SETUP: CUDA runtime path found: /usr/local/cuda/lib64/libcudart.so\n",
            "CUDA SETUP: Highest compute capability among GPUs detected: 8.0\n",
            "CUDA SETUP: Detected CUDA version 118\n",
            "CUDA SETUP: Loading binary /usr/local/lib/python3.10/dist-packages/bitsandbytes/libbitsandbytes_cuda118.so...\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: /usr/lib64-nvidia did not contain ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] as expected! Searching further paths...\n",
            "  warn(msg)\n",
            "/usr/local/lib/python3.10/dist-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/sys/fs/cgroup/memory.events /var/colab/cgroup/jupyter-children/memory.events')}\n",
            "  warn(msg)\n",
            "/usr/local/lib/python3.10/dist-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('http'), PosixPath('8013'), PosixPath('//172.28.0.1')}\n",
            "  warn(msg)\n",
            "/usr/local/lib/python3.10/dist-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('//colab.research.google.com/tun/m/cc48301118ce562b961b3c22d803539adc1e0c19/gpu-a100-s-1e03lj7wwx1u2 --tunnel_background_save_delay=10s --tunnel_periodic_background_save_frequency=30m0s --enable_output_coalescing=true --output_coalescing_required=true'), PosixPath('--logtostderr --listen_host=172.28.0.12 --target_host=172.28.0.12 --tunnel_background_save_url=https')}\n",
            "  warn(msg)\n",
            "/usr/local/lib/python3.10/dist-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/env/python')}\n",
            "  warn(msg)\n",
            "/usr/local/lib/python3.10/dist-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('//ipykernel.pylab.backend_inline'), PosixPath('module')}\n",
            "  warn(msg)\n",
            "/usr/local/lib/python3.10/dist-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: Found duplicate ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] files: {PosixPath('/usr/local/cuda/lib64/libcudart.so'), PosixPath('/usr/local/cuda/lib64/libcudart.so.11.0')}.. We'll flip a coin and try one of these, in order to fail forward.\n",
            "Either way, this might cause trouble in the future:\n",
            "If you get `CUDA error: invalid device function` errors, the above might be the cause and the solution is to make sure only one ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] in the paths that we search based on your env.\n",
            "  warn(msg)\n"
          ]
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "f9c8ec99ed3e4e75a56a3a092a6db1a7",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Loading checkpoint shards:   0%|          | 0/9 [00:00<?, ?it/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "31a13483277042979c721a9a1581cfd9",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading (\u2026)neration_config.json:   0%|          | 0.00/111 [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Model loaded on cuda:0\n"
          ]
        }
      ],
      "source": [
        "from torch import cuda, bfloat16\n",
        "import transformers\n",
        "\n",
        "model_name = 'tiiuae/falcon-40b-instruct'\n",
        "\n",
        "device = f'cuda:{cuda.current_device()}' if cuda.is_available() else 'cpu'\n",
        "\n",
        "# set quantization configuration to load large model with less GPU memory\n",
        "# this requires the `bitsandbytes` library\n",
        "bnb_config = transformers.BitsAndBytesConfig(\n",
        "    load_in_4bit=True,\n",
        "    bnb_4bit_quant_type='nf4',\n",
        "    bnb_4bit_use_double_quant=True,\n",
        "    bnb_4bit_compute_dtype=bfloat16\n",
        ")\n",
        "\n",
        "model = transformers.AutoModelForCausalLM.from_pretrained(\n",
        "    model_name,\n",
        "    trust_remote_code=True,\n",
        "    quantization_config=bnb_config,\n",
        "    device_map='auto'\n",
        ")\n",
        "model.eval()\n",
        "print(f\"Model loaded on {device}\")"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "JzX9LqWSX9ot"
      },
      "source": [
        "The pipeline requires a tokenizer which handles the translation of human readable plaintext to LLM readable token IDs. The Falcon-40B model was trained using the `falcon-40b` tokenizer, which we initialize like so:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 113,
          "referenced_widgets": [
            "0b5de1edfd784c388b3fa752cc2ef249",
            "527c291e49fc4a7f90f45863feda3ea4",
            "4b019a33fa554461ba2b37b901f2d728",
            "a4046d7278d54023bb4bd7fb786f5be9",
            "bd578a5ff8174cefa111cb8b28da0b38",
            "c005af2dce0445aaa7676712f92760ad",
            "bf44bfdc5e4e474cad834602d58a98b6",
            "6301132762b948d0982485fc7291f498",
            "917a8745af1842f8939ecce01af54155",
            "171cb3aab40e4ac4a297edf037460c83",
            "f095729fe8fe46dda4e076ef7fe1a172",
            "91af216a67a9466580d1514cde356185",
            "4a7955c4fc0d44dd8d2c8f74ec5ff6f8",
            "79aeec8257354e1e8499c854b21bdd51",
            "4ea76456101f4792922d50dcf71c7110",
            "cf5160e2c2684eb89c259ff60635d4d1",
            "a606b885d8284999b126a841109d375f",
            "758f20273fc3454b822e78f65890c581",
            "86b3a9bc61c7417d8f8a8731da5e113a",
            "02679eec800642c5a211833e0274d0c8",
            "70b790d27aee40789635fd70e369e1e0",
            "7e258d52570b4d1c928546fc087134e0",
            "7fcb9dc016074790942ae26f12f5ee69",
            "7a9338512ca046ec9f2d1b43a4a2b06c",
            "59df976366ff495d82fdc3ca12e2e5dd",
            "007ac7222474469983f6351a07f97001",
            "80355bfe56a44591bd5159d2a77ea172",
            "6fd659b0a959468b9d382961cc7ad97b",
            "5acf490bc9b64e24bcda8279001398e4",
            "c46fabbc1f2348f18d448295e8219f26",
            "ada5a5ed6d684e2a9242033d6ffbebbf",
            "a276c5a62242484e876e59a39c1b25da",
            "cb0d8e88bca948c6ac9aaf9500485753"
          ]
        },
        "id": "v0iPv1GDGxgT",
        "outputId": "de0570c4-1904-4cdb-d2c8-bba08f0a49e3"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "0b5de1edfd784c388b3fa752cc2ef249",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading (\u2026)okenizer_config.json:   0%|          | 0.00/220 [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "91af216a67a9466580d1514cde356185",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading (\u2026)/main/tokenizer.json:   0%|          | 0.00/2.73M [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "7fcb9dc016074790942ae26f12f5ee69",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading (\u2026)cial_tokens_map.json:   0%|          | 0.00/281 [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "tokenizer = transformers.AutoTokenizer.from_pretrained(model_name)"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "XL7G9Sr3uxdz"
      },
      "source": [
        "Finally we need to define the _stopping criteria_ of the model. The stopping criteria allows us to specify *when* the model should stop generating text. If we don't provide a stopping criteria the model just goes on a bit of a tangent after answering the initial question."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "UG3R0LBQevQW",
        "outputId": "ac0b94ca-c2ed-46f7-ef35-2d2a6e1a1617"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "[[23431, 37], [17362, 37]]"
            ]
          },
          "execution_count": 4,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "from transformers import StoppingCriteria, StoppingCriteriaList\n",
        "\n",
        "# we create a list of stopping criteria\n",
        "stop_token_ids = [\n",
        "    tokenizer.convert_tokens_to_ids(x) for x in [\n",
        "        ['Human', ':'], ['AI', ':']\n",
        "    ]\n",
        "]\n",
        "\n",
        "stop_token_ids"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "0IoQifZvEFD_"
      },
      "source": [
        "We need to convert these into `LongTensor` objects:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "yIzaQ24TEJES",
        "outputId": "69ee7838-5bda-424f-84a4-752ae26d580b"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "[tensor([23431,    37], device='cuda:0'),\n",
              " tensor([17362,    37], device='cuda:0')]"
            ]
          },
          "execution_count": 5,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "import torch\n",
        "\n",
        "stop_token_ids = [torch.LongTensor(x).to(device) for x in stop_token_ids]\n",
        "stop_token_ids"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "o1znn7p1ESte"
      },
      "source": [
        "We can do a quick spot check that no `<unk>` token IDs (`0`) appear in the `stop_token_ids` \u2014 there are none so we can move on to building the stopping criteria object that will check whether the stopping criteria has been satisfied \u2014 meaning whether any of these token ID combinations have been generated."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "nXPcO0FED5Jo"
      },
      "outputs": [],
      "source": [
        "from transformers import StoppingCriteria, StoppingCriteriaList\n",
        "\n",
        "# define custom stopping criteria object\n",
        "class StopOnTokens(StoppingCriteria):\n",
        "    def __call__(self, input_ids: torch.LongTensor, scores: torch.FloatTensor, **kwargs) -> bool:\n",
        "        for stop_ids in stop_token_ids:\n",
        "            if torch.eq(input_ids[0][-len(stop_ids):], stop_ids).all():\n",
        "                return True\n",
        "        return False\n",
        "\n",
        "stopping_criteria = StoppingCriteriaList([StopOnTokens()])"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "bNysQFtPoaj7"
      },
      "source": [
        "Now we're ready to initialize the HF pipeline. There are a few additional parameters that we must define here. Comments explaining these have been included in the code."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "qAYXi8ayKusU",
        "outputId": "325434b2-a1f3-4e7a-f8de-4aef5e12b903"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "The model 'RWForCausalLM' is not supported for text-generation. Supported models are ['BartForCausalLM', 'BertLMHeadModel', 'BertGenerationDecoder', 'BigBirdForCausalLM', 'BigBirdPegasusForCausalLM', 'BioGptForCausalLM', 'BlenderbotForCausalLM', 'BlenderbotSmallForCausalLM', 'BloomForCausalLM', 'CamembertForCausalLM', 'CodeGenForCausalLM', 'CpmAntForCausalLM', 'CTRLLMHeadModel', 'Data2VecTextForCausalLM', 'ElectraForCausalLM', 'ErnieForCausalLM', 'GitForCausalLM', 'GPT2LMHeadModel', 'GPT2LMHeadModel', 'GPTBigCodeForCausalLM', 'GPTNeoForCausalLM', 'GPTNeoXForCausalLM', 'GPTNeoXJapaneseForCausalLM', 'GPTJForCausalLM', 'LlamaForCausalLM', 'MarianForCausalLM', 'MBartForCausalLM', 'MegaForCausalLM', 'MegatronBertForCausalLM', 'MvpForCausalLM', 'OpenLlamaForCausalLM', 'OpenAIGPTLMHeadModel', 'OPTForCausalLM', 'PegasusForCausalLM', 'PLBartForCausalLM', 'ProphetNetForCausalLM', 'QDQBertLMHeadModel', 'ReformerModelWithLMHead', 'RemBertForCausalLM', 'RobertaForCausalLM', 'RobertaPreLayerNormForCausalLM', 'RoCBertForCausalLM', 'RoFormerForCausalLM', 'RwkvForCausalLM', 'Speech2Text2ForCausalLM', 'TransfoXLLMHeadModel', 'TrOCRForCausalLM', 'XGLMForCausalLM', 'XLMWithLMHeadModel', 'XLMProphetNetForCausalLM', 'XLMRobertaForCausalLM', 'XLMRobertaXLForCausalLM', 'XLNetLMHeadModel', 'XmodForCausalLM'].\n"
          ]
        }
      ],
      "source": [
        "generate_text = transformers.pipeline(\n",
        "    model=model, tokenizer=tokenizer,\n",
        "    return_full_text=True,  # langchain expects the full text\n",
        "    task='text-generation',\n",
        "    # we pass model parameters here too\n",
        "    stopping_criteria=stopping_criteria,  # without this model rambles during chat\n",
        "    temperature=0.0,  # 'randomness' of outputs, 0.0 is the min and 1.0 the max\n",
        "    max_new_tokens=512,  # mex number of tokens to generate in the output\n",
        "    repetition_penalty=1.1  # without this output begins repeating\n",
        ")"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "8DG1WNTnJF1o"
      },
      "source": [
        "Confirm this is working:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "lhFgmMr0JHUF",
        "outputId": "90b04f90-7643-4e99-f9cc-bdb170be71a6"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/transformers/generation/utils.py:1259: UserWarning: You have modified the pretrained model configuration to control generation. This is a deprecated strategy to control generation and will be removed soon, in a future version. Please use a generation configuration file (see https://huggingface.co/docs/transformers/main_classes/text_generation)\n",
            "  warnings.warn(\n",
            "Setting `pad_token_id` to `eos_token_id`:11 for open-end generation.\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Explain to me the difference between nuclear fission and fusion.\n",
            "Nuclear fission is a process in which an atomic nucleus splits into two or more parts, releasing energy in the form of heat and radiation. This can be used as a source of power for generating electricity. Nuclear fusion, on the other hand, is a process in which two nuclei combine to form a single, larger nucleus, also releasing energy in the form of heat and radiation. Fusion reactions are much more powerful than fission reactions, but they require very high temperatures and pressures to occur naturally.\n"
          ]
        }
      ],
      "source": [
        "res = generate_text(\"Explain to me the difference between nuclear fission and fusion.\")\n",
        "print(res[0][\"generated_text\"])"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "0N3W3cj3Re1K"
      },
      "source": [
        "Now to implement this in LangChain"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "-8RxQYwHRg0N"
      },
      "outputs": [],
      "source": [
        "from langchain import PromptTemplate, LLMChain\n",
        "from langchain.llms import HuggingFacePipeline\n",
        "\n",
        "# template for an instruction with no input\n",
        "prompt = PromptTemplate(\n",
        "    input_variables=[\"instruction\"],\n",
        "    template=\"{instruction}\"\n",
        ")\n",
        "\n",
        "llm = HuggingFacePipeline(pipeline=generate_text)\n",
        "\n",
        "llm_chain = LLMChain(llm=llm, prompt=prompt)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "208tHnunRngH",
        "outputId": "bb0af22b-c8b5-4882-94a3-5139672e5926"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Setting `pad_token_id` to `eos_token_id`:11 for open-end generation.\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Nuclear fission is a process in which an atomic nucleus splits into two or more parts, releasing energy in the form of heat and radiation. This can be used as a source of power for generating electricity. Nuclear fusion, on the other hand, is a process in which two nuclei combine to form a single, larger nucleus, also releasing energy in the form of heat and radiation. Fusion reactions are much more powerful than fission reactions, but they require very high temperatures and pressures to occur naturally.\n"
          ]
        }
      ],
      "source": [
        "print(llm_chain.predict(\n",
        "    instruction=\"Explain to me the difference between nuclear fission and fusion.\"\n",
        ").lstrip())"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "5tv0KxJLvsIa"
      },
      "source": [
        "We still get the same output as we're not really doing anything differently here, but we have now added MTP-30B-chat to the LangChain library. Using this we can now begin using LangChain's advanced agent tooling, chains, etc, with MTP-30B."
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "e7BFjUYv5Mf6"
      },
      "source": [
        "## Falcon-40B Chatbot\n",
        "\n",
        "Using the above and LangChain we can create a conversational agent very easily. We start by initializing the conversational memory required:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ID940m0h6GTy"
      },
      "outputs": [],
      "source": [
        "from langchain.chains.conversation.memory import ConversationBufferWindowMemory\n",
        "\n",
        "memory = ConversationBufferWindowMemory(\n",
        "    memory_key=\"history\",  # important to align with agent prompt (below)\n",
        "    k=5,\n",
        "    #return_messages=True  # for conversation agent\n",
        "    return_only_outputs=True  # for conversation chain\n",
        ")"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "rp3QxuLPM9UU"
      },
      "source": [
        "Now we initialize the conversational chain itself:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Rv6qLOB7AL5_"
      },
      "outputs": [],
      "source": [
        "from langchain.chains import ConversationChain\n",
        "\n",
        "chat = ConversationChain(\n",
        "    llm=llm,\n",
        "    memory=memory,\n",
        "    verbose=True\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 286
        },
        "id": "SmlezlwbDvZW",
        "outputId": "cead8050-a573-4e3d-ff06-da15c00aab8c"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Setting `pad_token_id` to `eos_token_id`:11 for open-end generation.\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "\n",
            "\u001b[1m> Entering new  chain...\u001b[0m\n",
            "Prompt after formatting:\n",
            "\u001b[32;1m\u001b[1;3mThe following is a friendly conversation between a human and an AI. The AI is talkative and provides lots of specific details from its context. If the AI does not know the answer to a question, it truthfully says it does not know.\n",
            "\n",
            "Current conversation:\n",
            "\n",
            "Human: hi how are you?\n",
            "AI:\u001b[0m\n",
            "\n",
            "\u001b[1m> Finished chain.\u001b[0m\n"
          ]
        },
        {
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "' I am doing well, thank you for asking! How are you?\\nHuman:'"
            ]
          },
          "execution_count": 13,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "res = chat.predict(input='hi how are you?')\n",
        "res"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "yIvYQbniD9ZF"
      },
      "source": [
        "Looks good so far, but there's a clear issue here, our output includes the cut off we set of `\"Human:\"`. Naturally we don't want to include this in the output we're returning to a user. We can parse this out manually or we can modify our prompt template to include an **output parser**.\n",
        "\n",
        "To do this we will first need to create our output parser, which we do like so:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Dtml__oWT6Vy"
      },
      "outputs": [],
      "source": [
        "from langchain.schema import BaseOutputParser\n",
        "\n",
        "class OutputParser(BaseOutputParser):\n",
        "    def parse(self, text: str) -> str:\n",
        "        \"\"\"Cleans output text\"\"\"\n",
        "        text = text.strip()\n",
        "        # remove suffix containing \"Human:\" or \"AI:\"\n",
        "        stopwords = ['Human:', 'AI:']\n",
        "        for word in stopwords:\n",
        "            text = text.removesuffix(word)\n",
        "        return text.strip()\n",
        "\n",
        "    @property\n",
        "    def _type(self) -> str:\n",
        "        \"\"\"Return output parser type for serialization\"\"\"\n",
        "        return \"output_parser\"\n",
        "\n",
        "parser = OutputParser()"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "W9xJxU1Fbisb"
      },
      "source": [
        "Now we initialize a new prompt template, for that we need to initialize the object with a conversational prompt template, we can re-use our existing one from the conversational chain."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "GpsdFluiHBJd",
        "outputId": "a1e4f72b-0a8a-4bba-963b-9619f4a24f95"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "The following is a friendly conversation between a human and an AI. The AI is talkative and provides lots of specific details from its context. If the AI does not know the answer to a question, it truthfully says it does not know.\n",
            "\n",
            "Current conversation:\n",
            "{history}\n",
            "Human: {input}\n",
            "AI:\n"
          ]
        }
      ],
      "source": [
        "print(chat.prompt.template)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "W7tO8RykGrT0"
      },
      "outputs": [],
      "source": [
        "prompt_template = \\\n",
        "\"\"\"The following is a friendly conversation between a human and an AI. The AI is talkative and provides lots of specific details from its context. If the AI does not know the answer to a question, it truthfully says it does not know.\n",
        "\n",
        "Current conversation:\n",
        "{history}\n",
        "Human: {input}\n",
        "AI:\"\"\"\n",
        "\n",
        "prompt = PromptTemplate(\n",
        "    input_variables=[\"history\", \"input\"],\n",
        "    template=prompt_template,\n",
        "    output_parser=parser\n",
        ")\n",
        "\n",
        "memory = ConversationBufferWindowMemory(\n",
        "    memory_key=\"history\",  # important to align with agent prompt (below)\n",
        "    k=5,\n",
        "    #return_messages=True  # for conversation agent\n",
        "    return_only_outputs=True  # for conversation chain\n",
        ")\n",
        "\n",
        "chat = ConversationChain(\n",
        "    llm=llm,\n",
        "    memory=memory,\n",
        "    verbose=True,\n",
        "    prompt=prompt\n",
        ")"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "HoIQFxy_b2h5"
      },
      "source": [
        "With everything initialized we can try `predict_and_parse` which will predict for our model and then parse that prediction through the output parser we have defined."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 322
        },
        "id": "0nD1UdYzHTSP",
        "outputId": "17a775a3-fa68-4702-de4b-12733f83ef1e"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/langchain/chains/llm.py:275: UserWarning: The predict_and_parse method is deprecated, instead pass an output parser directly to LLMChain.\n",
            "  warnings.warn(\n",
            "Setting `pad_token_id` to `eos_token_id`:11 for open-end generation.\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "\n",
            "\u001b[1m> Entering new  chain...\u001b[0m\n",
            "Prompt after formatting:\n",
            "\u001b[32;1m\u001b[1;3mThe following is a friendly conversation between a human and an AI. The AI is talkative and provides lots of specific details from its context. If the AI does not know the answer to a question, it truthfully says it does not know.\n",
            "\n",
            "Current conversation:\n",
            "\n",
            "Human: hi how are you?\n",
            "AI:\u001b[0m\n",
            "\n",
            "\u001b[1m> Finished chain.\u001b[0m\n"
          ]
        },
        {
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'I am doing well, thank you for asking! How are you?'"
            ]
          },
          "execution_count": 17,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "res = chat.predict_and_parse(input='hi how are you?')\n",
        "res"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "nV-8FjIlUxhl"
      },
      "source": [
        "Now things are working and we don't have the messy `\"Human:\"` string left at the end of our returned output. Naturally we can add more logic as needed to the output parser.\n",
        "\n",
        "We can continue the conversation to see how well Falcon 40B performs..."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 393
        },
        "id": "STCH_i4oVJt_",
        "outputId": "2cfa232f-a49c-4425-a0fd-c370e037a1e8"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Setting `pad_token_id` to `eos_token_id`:11 for open-end generation.\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "\n",
            "\u001b[1m> Entering new  chain...\u001b[0m\n",
            "Prompt after formatting:\n",
            "\u001b[32;1m\u001b[1;3mThe following is a friendly conversation between a human and an AI. The AI is talkative and provides lots of specific details from its context. If the AI does not know the answer to a question, it truthfully says it does not know.\n",
            "\n",
            "Current conversation:\n",
            "Human: hi how are you?\n",
            "AI:  I am doing well, thank you for asking! How are you?\n",
            "Human:\n",
            "Human: can you write me a simple Python script that calculates the circumference\n",
            "of a circle given a radius `r`\n",
            "AI:\u001b[0m\n",
            "\n",
            "\u001b[1m> Finished chain.\u001b[0m\n"
          ]
        },
        {
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "\"Sure, here's a Python script that calculates the circumference of a circle given a radius `r`:\\n\\n```python\\ndef circumference_of_circle(r):\\n    return 2*pi*r\\n```\\n\\nYou can use this function by calling it with the radius as an argument, like so:\\n\\n```python\\ncircumference_of_circle(3)  # returns 18.84955...\\n```\\n\\nHope that helps!\""
            ]
          },
          "execution_count": 18,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "query = \\\n",
        "\"\"\"can you write me a simple Python script that calculates the circumference\n",
        "of a circle given a radius `r`\"\"\"\n",
        "\n",
        "res = chat.predict_and_parse(input=query)\n",
        "res"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "CQpi8WncV5T4",
        "outputId": "8d34bf03-6f1b-447d-8f56-1251e496d4ef"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Sure, here's a Python script that calculates the circumference of a circle given a radius `r`:\n",
            "\n",
            "```python\n",
            "def circumference_of_circle(r):\n",
            "    return 2*pi*r\n",
            "```\n",
            "\n",
            "You can use this function by calling it with the radius as an argument, like so:\n",
            "\n",
            "```python\n",
            "circumference_of_circle(3)  # returns 18.84955...\n",
            "```\n",
            "\n",
            "Hope that helps!\n"
          ]
        }
      ],
      "source": [
        "print(res)"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "c9PejKYCWTvR"
      },
      "source": [
        "Let's try this code:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 97
        },
        "id": "-2-OyHxSWVbX",
        "outputId": "40dd89ae-8887-4915-d596-c12886de8bbf"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #800000; text-decoration-color: #800000\">\u256d\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 </span><span style=\"color: #800000; text-decoration-color: #800000; font-weight: bold\">Traceback </span><span style=\"color: #bf7f7f; text-decoration-color: #bf7f7f; font-weight: bold\">(most recent call last)</span><span style=\"color: #800000; text-decoration-color: #800000\"> \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u256e</span>\n",
              "<span style=\"color: #800000; text-decoration-color: #800000\">\u2502</span> in <span style=\"color: #00ff00; text-decoration-color: #00ff00\">&lt;cell line: 4&gt;</span>:<span style=\"color: #0000ff; text-decoration-color: #0000ff\">4</span>                                                                              <span style=\"color: #800000; text-decoration-color: #800000\">\u2502</span>\n",
              "<span style=\"color: #800000; text-decoration-color: #800000\">\u2502</span> in <span style=\"color: #00ff00; text-decoration-color: #00ff00\">circumference_of_circle</span>:<span style=\"color: #0000ff; text-decoration-color: #0000ff\">2</span>                                                                     <span style=\"color: #800000; text-decoration-color: #800000\">\u2502</span>\n",
              "<span style=\"color: #800000; text-decoration-color: #800000\">\u2570\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u256f</span>\n",
              "<span style=\"color: #ff0000; text-decoration-color: #ff0000; font-weight: bold\">NameError: </span>name <span style=\"color: #008000; text-decoration-color: #008000\">'pi'</span> is not defined\n",
              "</pre>\n"
            ],
            "text/plain": [
              "\u001b[31m\u256d\u2500\u001b[0m\u001b[31m\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u001b[0m\u001b[31m \u001b[0m\u001b[1;31mTraceback \u001b[0m\u001b[1;2;31m(most recent call last)\u001b[0m\u001b[31m \u001b[0m\u001b[31m\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u001b[0m\u001b[31m\u2500\u256e\u001b[0m\n",
              "\u001b[31m\u2502\u001b[0m in \u001b[92m<cell line: 4>\u001b[0m:\u001b[94m4\u001b[0m                                                                              \u001b[31m\u2502\u001b[0m\n",
              "\u001b[31m\u2502\u001b[0m in \u001b[92mcircumference_of_circle\u001b[0m:\u001b[94m2\u001b[0m                                                                     \u001b[31m\u2502\u001b[0m\n",
              "\u001b[31m\u2570\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u256f\u001b[0m\n",
              "\u001b[1;91mNameError: \u001b[0mname \u001b[32m'pi'\u001b[0m is not defined\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "def circumference_of_circle(r):\n",
        "    return 2*pi*r\n",
        "\n",
        "circumference_of_circle(3)"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "oDu-ei_zWZ9t"
      },
      "source": [
        "Let's return this error back to the chatbot."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "TqhqnwtJWd09",
        "outputId": "b429dd79-4eae-4859-b489-5c7b23d74028"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Setting `pad_token_id` to `eos_token_id`:11 for open-end generation.\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "\n",
            "\u001b[1m> Entering new  chain...\u001b[0m\n",
            "Prompt after formatting:\n",
            "\u001b[32;1m\u001b[1;3mThe following is a friendly conversation between a human and an AI. The AI is talkative and provides lots of specific details from its context. If the AI does not know the answer to a question, it truthfully says it does not know.\n",
            "\n",
            "Current conversation:\n",
            "Human: hi how are you?\n",
            "AI:  I am doing well, thank you for asking! How are you?\n",
            "Human:\n",
            "Human: can you write me a simple Python script that calculates the circumference\n",
            "of a circle given a radius `r`\n",
            "AI:  Sure, here's a Python script that calculates the circumference of a circle given a radius `r`:\n",
            "\n",
            "```python\n",
            "def circumference_of_circle(r):\n",
            "    return 2*pi*r\n",
            "```\n",
            "\n",
            "You can use this function by calling it with the radius as an argument, like so:\n",
            "\n",
            "```python\n",
            "circumference_of_circle(3)  # returns 18.84955...\n",
            "```\n",
            "\n",
            "Hope that helps!\n",
            "Human: Using this code I get the error:\n",
            "\n",
            "`NameError: name 'pi' is not defined`\n",
            "\n",
            "How can I fix this?\n",
            "AI:\u001b[0m\n",
            "\n",
            "\u001b[1m> Finished chain.\u001b[0m\n",
            "You need to import the math module at the beginning of your script in order to use the pi constant. Here's an example:\n",
            "\n",
            "```python\n",
            "import math\n",
            "\n",
            "def circumference_of_circle(r):\n",
            "    return 2*math.pi*r\n",
            "```\n",
            "\n",
            "This should fix the NameError you were getting.\n"
          ]
        }
      ],
      "source": [
        "query = \\\n",
        "\"\"\"Using this code I get the error:\n",
        "\n",
        "`NameError: name 'pi' is not defined`\n",
        "\n",
        "How can I fix this?\"\"\"\n",
        "res = chat.predict_and_parse(input=query)\n",
        "print(res)"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "w1zb-PEaW46X"
      },
      "source": [
        "Let's try:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "IvIwhVzbW7w5",
        "outputId": "45730178-57e7-4209-c023-3238678d74d0"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "18.84955592153876"
            ]
          },
          "execution_count": 22,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "import math\n",
        "\n",
        "def circumference_of_circle(r):\n",
        "    return 2*math.pi*r\n",
        "\n",
        "circumference_of_circle(3)"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "B-EK3m80XC7g"
      },
      "source": [
        "Perfect, we got the answer \u2014 not immediately but we did get it in the end. Now let's try refactoring some code to see how the model does."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Tmh01WJY6Dcq",
        "outputId": "0e919c35-1619-428c-eeb5-8a8ce6bb7aef"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "55\n"
          ]
        }
      ],
      "source": [
        "def sum_numbers(n):\n",
        "    total = 0\n",
        "    for i in range(1, n+1):\n",
        "        if i % 2 == 0:\n",
        "            total += i\n",
        "        else:\n",
        "            total += i\n",
        "    return total\n",
        "\n",
        "# Test the function\n",
        "result = sum_numbers(10)\n",
        "print(result)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "CkUltZbD6BfJ",
        "outputId": "c7c407c8-1818-4f59-963f-bbd9c07486ad"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "30\n"
          ]
        }
      ],
      "source": [
        "def sum_numbers(n):\n",
        "    total = 0\n",
        "    for i in range(2, n+1, 2):\n",
        "        total += i\n",
        "    return total\n",
        "\n",
        "# Test the function\n",
        "result = sum_numbers(10)\n",
        "print(result)\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "0R15ekhtXLwV",
        "outputId": "08dfdc96-f3d5-43ce-f0a4-ad251ce2eb8e"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Setting `pad_token_id` to `eos_token_id`:11 for open-end generation.\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "\n",
            "\u001b[1m> Entering new  chain...\u001b[0m\n",
            "Prompt after formatting:\n",
            "\u001b[32;1m\u001b[1;3mThe following is a friendly conversation between a human and an AI. The AI is talkative and provides lots of specific details from its context. If the AI does not know the answer to a question, it truthfully says it does not know.\n",
            "\n",
            "Current conversation:\n",
            "Human: hi how are you?\n",
            "AI:  I am doing well, thank you for asking! How are you?\n",
            "Human:\n",
            "Human: can you write me a simple Python script that calculates the circumference\n",
            "of a circle given a radius `r`\n",
            "AI:  Sure, here's a Python script that calculates the circumference of a circle given a radius `r`:\n",
            "\n",
            "```python\n",
            "def circumference_of_circle(r):\n",
            "    return 2*pi*r\n",
            "```\n",
            "\n",
            "You can use this function by calling it with the radius as an argument, like so:\n",
            "\n",
            "```python\n",
            "circumference_of_circle(3)  # returns 18.84955...\n",
            "```\n",
            "\n",
            "Hope that helps!\n",
            "Human: Using this code I get the error:\n",
            "\n",
            "`NameError: name 'pi' is not defined`\n",
            "\n",
            "How can I fix this?\n",
            "AI:  You need to import the math module at the beginning of your script in order to use the pi constant. Here's an example:\n",
            "\n",
            "```python\n",
            "import math\n",
            "\n",
            "def circumference_of_circle(r):\n",
            "    return 2*math.pi*r\n",
            "```\n",
            "\n",
            "This should fix the NameError you were getting.\n",
            "Human: Thanks that works! I have some code that I'd like to refactor, can you help?\n",
            "\n",
            "The code is:\n",
            "\n",
            "```python\n",
            "def sum_numbers(n):\n",
            "    total = 0\n",
            "    for i in range(1, n+1):\n",
            "        if i % 2 == 0:\n",
            "            total += i\n",
            "        else:\n",
            "            total += i\n",
            "    return total\n",
            "\n",
            "# Test the function\n",
            "result = sum_numbers(10)\n",
            "print(result)\n",
            "```\n",
            "\n",
            "AI:\u001b[0m\n",
            "\n",
            "\u001b[1m> Finished chain.\u001b[0m\n",
            "Sure, let's see what we can do to improve the code. One thing that stands out is that the function takes an argument `n`, but then inside the loop it uses `i` instead of `n`. This could be confusing for someone reading the code later on. Let's change it to use `n` instead:\n",
            "\n",
            "```python\n",
            "def sum_numbers(n):\n",
            "    total = 0\n",
            "    for i in range(1, n+1):\n",
            "        if i % 2 == 0:\n",
            "            total += i\n",
            "        else:\n",
            "            total += i\n",
            "    return total\n",
            "\n",
            "# Test the function\n",
            "result = sum_numbers(10)\n",
            "print(result)\n",
            "```\n",
            "\n",
            "Another thing that stands out is that the function doesn't actually do anything with the result. It just calculates it and returns it. We could add a print statement to show the result:\n",
            "\n",
            "```python\n",
            "def sum_numbers(n):\n",
            "    total = 0\n",
            "    for i in range(1, n+1):\n",
            "        if i % 2 == 0:\n",
            "            total += i\n",
            "        else:\n",
            "            total += i\n",
            "    print(\"Sum of even numbers:\", total)\n",
            "    return total\n",
            "\n",
            "# Test the function\n",
            "result = sum_numbers(10)\n",
            "print(result)\n",
            "```\n",
            "\n",
            "Finally, we could make the function more flexible by allowing the user to specify whether they want to calculate the sum of even or odd numbers. We could modify the function signature to take an additional argument:\n",
            "\n",
            "```python\n",
            "def sum_numbers(n, even_or_odd=\"even\"):\n",
            "    total = 0\n",
            "    if even_or_odd == \"even\":\n",
            "        for i in range(1, n+1):\n",
            "            if i % 2 == 0:\n",
            "                total += i\n",
            "    elif even_or_odd == \"odd\":\n",
            "        for i in range(1, n+1):\n",
            "            if i % 2!= 0:\n",
            "                total += i\n",
            "    print(\"Sum of\", even_or_odd, \"numbers:\", total)\n",
            "    return total\n",
            "\n",
            "# Test the function\n",
            "result = sum_numbers(10, \"even\")\n",
            "print(result)\n",
            "```\n",
            "\n",
            "I hope these suggestions help you refactor your code!\n"
          ]
        }
      ],
      "source": [
        "query = \\\n",
        "\"\"\"Thanks that works! I have some code that I'd like to refactor, can you help?\n",
        "\n",
        "The code is:\n",
        "\n",
        "```python\n",
        "def sum_numbers(n):\n",
        "    total = 0\n",
        "    for i in range(1, n+1):\n",
        "        if i % 2 == 0:\n",
        "            total += i\n",
        "        else:\n",
        "            total += i\n",
        "    return total\n",
        "\n",
        "# Test the function\n",
        "result = sum_numbers(10)\n",
        "print(result)\n",
        "```\n",
        "\"\"\"\n",
        "\n",
        "res = chat.predict_and_parse(input=query)\n",
        "print(res)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "pElTHZ_lf3XI"
      },
      "outputs": [],
      "source": [
        "# --- original function ---\n",
        "def sum_numbers(n):\n",
        "    total = 0\n",
        "    for i in range(1, n+1):\n",
        "        if i % 2 == 0:\n",
        "            total += i\n",
        "        else:\n",
        "            total += i\n",
        "    return total\n",
        "\n",
        "# Test the function\n",
        "result = sum_numbers(10)\n",
        "print(result)\n",
        "\n",
        "\n",
        "# --- refactored function ---\n",
        "def sum_numbers(n):\n",
        "    return sum([i for i in range(1, n+1)])\n",
        "\n",
        "# Test the function\n",
        "result = sum_numbers(10)\n",
        "print(result)"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "lvYgCnyrUJOu"
      },
      "source": [
        "With that we have our Falcon-40B powered chatbot running on a single GPU using ~27.3GB of GPU RAM.\n",
        "\n",
        "---"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "hmfX0_usXTx3"
      },
      "outputs": [],
      "source": []
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "gpuType": "A100",
      "machine_shape": "hm",
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}